{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Creating Sequence to Sequence Models\n",
    "\n",
    "----------------------------------\n",
    "\n",
    "Here we show how to implement sequence to sequence models. Specifically, we will build an English to German translation model.\n",
    "\n",
    "\n",
    "We start by loading the necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import sys\n",
    "import string\n",
    "import requests\n",
    "import io\n",
    "import numpy as np\n",
    "import collections\n",
    "import random\n",
    "import pickle\n",
    "import string\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "from zipfile import ZipFile\n",
    "from collections import Counter\n",
    "from tensorflow.python.framework import ops\n",
    "ops.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "TensorFlow has a separate `models` folder that holds some great models that we can use.  One of these is a seq2seq model that we will use to train on our translation data. The models are located here:\n",
    "\n",
    "https://github.com/tensorflow/models\n",
    "\n",
    "And specifically, the seq2seq model and utilities are located here:\n",
    "\n",
    "https://github.com/tensorflow/models/tree/master/tutorials/rnn/translate\n",
    "    \n",
    "If you want to download them separately, you may do that.  The goal here is to access the `data_utils.py` and the `seq2seq_model.py` scripts in the above folder.  We will import them.\n",
    "\n",
    "The following block of code will import the whole `models` repository into the temp folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "local_repository = 'temp'\n",
    "if not os.path.exists(local_repository):\n",
    "    from git import Repo\n",
    "    tf_model_repository = 'https://github.com/tensorflow/models'\n",
    "    Repo.clone_from(tf_model_repository, local_repository)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that we have downloaded the model files, we import the necessary funtions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sys.path.insert(0, 'temp/tutorials/rnn/translate/')\n",
    "import seq2seq_model as seq2seq_model\n",
    "import data_utils as data_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we start a computational graph session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We setup the model parameters for our seq2seq model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Model Parameters\n",
    "learning_rate = 0.1\n",
    "lr_decay_rate = 0.99\n",
    "lr_decay_every = 100\n",
    "max_gradient = 5.0\n",
    "batch_size = 50\n",
    "num_layers = 3\n",
    "rnn_size = 500\n",
    "layer_size = 512\n",
    "generations = 10000\n",
    "vocab_size = 10000\n",
    "save_every = 1000\n",
    "eval_every = 500\n",
    "output_every = 50\n",
    "punct = string.punctuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We want to download the data, save the data, save the model, and create test english sentences for the model to be evaluated on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Data Parameters\n",
    "data_dir = 'temp'\n",
    "data_file = 'eng_ger.txt'\n",
    "model_path = 'seq2seq_model'\n",
    "full_model_dir = os.path.join(data_dir, model_path)\n",
    "\n",
    "# Test Translation from English (lowercase, no punct)\n",
    "test_english = ['hello where is my computer',\n",
    "                'the quick brown fox jumped over the lazy dog',\n",
    "                'is it going to rain tomorrow']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we create the model directory if it doesn't exist.\n",
    "\n",
    "After, we load the english-german sentence translation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading English-German Data\n"
     ]
    }
   ],
   "source": [
    "# Make Model Directory\n",
    "if not os.path.exists(full_model_dir):\n",
    "    os.makedirs(full_model_dir)\n",
    "\n",
    "print('Loading English-German Data')\n",
    "# Check for data, if it doesn't exist, download it and save it\n",
    "if not os.path.isfile(os.path.join(data_dir, data_file)):\n",
    "    print('Data not found, downloading Eng-Ger sentences from www.manythings.org')\n",
    "    sentence_url = 'http://www.manythings.org/anki/deu-eng.zip'\n",
    "    r = requests.get(sentence_url)\n",
    "    z = ZipFile(io.BytesIO(r.content))\n",
    "    file = z.read('deu.txt')\n",
    "    # Format Data\n",
    "    eng_ger_data = file.decode()\n",
    "    eng_ger_data = eng_ger_data.encode('ascii',errors='ignore')\n",
    "    eng_ger_data = eng_ger_data.decode().split('\\n')\n",
    "    # Write to file\n",
    "    with open(os.path.join(data_dir, data_file), 'w') as out_conn:\n",
    "        for sentence in eng_ger_data:\n",
    "            out_conn.write(sentence + '\\n')\n",
    "else:\n",
    "    eng_ger_data = []\n",
    "    with open(os.path.join(data_dir, data_file), 'r') as in_conn:\n",
    "        for row in in_conn:\n",
    "            eng_ger_data.append(row[:-1])\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Remove punctuation and separate the english-german sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Remove punctuation\n",
    "eng_ger_data = [''.join(char for char in sent if char not in punct) for sent in eng_ger_data]\n",
    "\n",
    "# Split each sentence by tabs    \n",
    "eng_ger_data = [x.split('\\t') for x in eng_ger_data if len(x)>=1]\n",
    "[english_sentence, german_sentence] = [list(x) for x in zip(*eng_ger_data)]\n",
    "english_sentence = [x.lower().split() for x in english_sentence]\n",
    "german_sentence = [x.lower().split() for x in german_sentence]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we process the english, german, and test sentences.  We need to create a lookup that translates words to an index number.  Note that we purposely exclude the index 0, because that will stand for an unknown word (rare or new words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing the vocabularies.\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "print('Processing the vocabularies.')\n",
    "# Process the English Vocabulary\n",
    "all_english_words = [word for sentence in english_sentence for word in sentence]\n",
    "all_english_counts = Counter(all_english_words)\n",
    "eng_word_keys = [x[0] for x in all_english_counts.most_common(vocab_size-1)] #-1 because 0=unknown is also in there\n",
    "eng_vocab2ix = dict(zip(eng_word_keys, range(1,vocab_size)))\n",
    "eng_ix2vocab = {val:key for key, val in eng_vocab2ix.items()}\n",
    "english_processed = []\n",
    "for sent in english_sentence:\n",
    "    temp_sentence = []\n",
    "    for word in sent:\n",
    "        try:\n",
    "            temp_sentence.append(eng_vocab2ix[word])\n",
    "        except:\n",
    "            temp_sentence.append(0)\n",
    "    english_processed.append(temp_sentence)\n",
    "\n",
    "\n",
    "# Process the German Vocabulary\n",
    "all_german_words = [word for sentence in german_sentence for word in sentence]\n",
    "all_german_counts = Counter(all_german_words)\n",
    "ger_word_keys = [x[0] for x in all_german_counts.most_common(vocab_size-1)]\n",
    "ger_vocab2ix = dict(zip(ger_word_keys, range(1,vocab_size)))\n",
    "ger_ix2vocab = {val:key for key, val in ger_vocab2ix.items()}\n",
    "german_processed = []\n",
    "for sent in german_sentence:\n",
    "    temp_sentence = []\n",
    "    for word in sent:\n",
    "        try:\n",
    "            temp_sentence.append(ger_vocab2ix[word])\n",
    "        except:\n",
    "            temp_sentence.append(0)\n",
    "    german_processed.append(temp_sentence)\n",
    "\n",
    "\n",
    "# Process the test english sentences, use '0' if word not in our vocab\n",
    "test_data = []\n",
    "for sentence in test_english:\n",
    "    temp_sentence = []\n",
    "    for word in sentence.split(' '):\n",
    "        try:\n",
    "            temp_sentence.append(eng_vocab2ix[word])\n",
    "        except:\n",
    "            # Use '0' if the word isn't in our vocabulary\n",
    "            temp_sentence.append(0)\n",
    "    test_data.append(temp_sentence)\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Because the input/output sentences can be of very different lengths (1 word vs a whole sentence), we will bucket the sentences by input/output lenghts.  For each bucket, we will train a different seq2seq model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data pts in bucket 0: 57222\n",
      "Data pts in bucket 1: 47613\n",
      "Data pts in bucket 2: 36158\n",
      "Data pts in bucket 3: 6795\n"
     ]
    }
   ],
   "source": [
    "# Define Buckets for sequence lengths\n",
    "# We will split data into the corresponding buckets:\n",
    "# (x1, y1), (x2, y2), ...\n",
    "# Where all entries in bucket 1: len(x)<x1 and len(y)<y1 and so on.\n",
    "x_maxs = [5, 7, 11, 50]\n",
    "y_maxs = [10, 12, 17, 60]\n",
    "buckets = [x for x in zip(x_maxs, y_maxs)]\n",
    "bucketed_data = [[] for _ in range(len(x_maxs))]\n",
    "for eng, ger in zip(english_processed, german_processed):\n",
    "    for ix, (x_max, y_max) in enumerate(zip(x_maxs, y_maxs)):\n",
    "        if (len(eng) <= x_max) and (len(ger) <= y_max):\n",
    "            bucketed_data[ix].append([eng, ger])\n",
    "            break\n",
    "\n",
    "# Print summaries of buckets\n",
    "train_bucket_sizes = [len(bucketed_data[b]) for b in range(len(buckets))]\n",
    "train_total_size = float(sum(train_bucket_sizes))\n",
    "for ix, bucket in enumerate(bucketed_data):\n",
    "    print('Data pts in bucket {}: {}'.format(ix, len(bucket)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we declare our seq2seq model function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Create sequence to sequence model\n",
    "def translation_model(sess, input_vocab_size, output_vocab_size,\n",
    "                      buckets, rnn_size, num_layers, max_gradient,\n",
    "                      learning_rate, lr_decay_rate, forward_only):\n",
    "    model = seq2seq_model.Seq2SeqModel(\n",
    "          input_vocab_size,\n",
    "          output_vocab_size,\n",
    "          buckets,\n",
    "          rnn_size,\n",
    "          num_layers,\n",
    "          max_gradient,\n",
    "          batch_size,\n",
    "          learning_rate,\n",
    "          lr_decay_rate,\n",
    "          forward_only=forward_only,\n",
    "          dtype=tf.float32)\n",
    "    return(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we create our train/test models. We want these models to use the exact same variables, so we have to tell TensorFlow to reuse the variables when we create the test-model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Translation Model\n"
     ]
    }
   ],
   "source": [
    "print('Creating Translation Model')\n",
    "input_vocab_size = vocab_size\n",
    "output_vocab_size = vocab_size\n",
    "\n",
    "translate_model = translation_model(sess=sess,\n",
    "                                    input_vocab_size=vocab_size,\n",
    "                                    output_vocab_size=vocab_size,\n",
    "                                    buckets=buckets,\n",
    "                                    rnn_size=rnn_size,\n",
    "                                    num_layers=num_layers,\n",
    "                                    max_gradient=max_gradient,\n",
    "                                    learning_rate=learning_rate,\n",
    "                                    lr_decay_rate=lr_decay_rate,\n",
    "                                    forward_only=False)\n",
    "\n",
    "# Tell TensorFlow to reuse the variables for the test model\n",
    "with tf.variable_scope(tf.get_variable_scope(), reuse=True):\n",
    "    #Reuse the variables for the test model\n",
    "    test_model = translation_model(sess=sess,\n",
    "                                    input_vocab_size=vocab_size,\n",
    "                                    output_vocab_size=vocab_size,\n",
    "                                    buckets=buckets,\n",
    "                                    rnn_size=rnn_size,\n",
    "                                    num_layers=num_layers,\n",
    "                                    max_gradient=max_gradient,\n",
    "                                    learning_rate=learning_rate,\n",
    "                                    lr_decay_rate=lr_decay_rate,\n",
    "                                    forward_only=True)\n",
    "    test_model.batch_size = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we can initialize the model variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Initialize all model variables\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We now start the training.  For each iteration, we pick a random bicket, and get a batch of sentences from that bucket.  There are then trained on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gen #50 out of 10000. Loss: 8.335\n",
      "Gen #100 out of 10000. Loss: 5.929\n",
      "Gen #150 out of 10000. Loss: 5.74\n",
      "Gen #200 out of 10000. Loss: 5.884\n",
      "Gen #250 out of 10000. Loss: 5.682\n",
      "Gen #300 out of 10000. Loss: 5.622\n",
      "Gen #350 out of 10000. Loss: 5.546\n",
      "Gen #400 out of 10000. Loss: 5.555\n",
      "Gen #450 out of 10000. Loss: 5.608\n",
      "Gen #500 out of 10000. Loss: 5.359\n",
      "English: hello where is my computer\n",
      "German: ['ich', 'ist', 'ist', 'ist', 'nicht', 'nicht', 'nicht', 'zu', 'zu', 'zu']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['wir', 'ist', 'ist', 'ist', 'ist', 'nicht', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['wir', 'ist', 'ist', 'ist', 'nicht', 'nicht', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #550 out of 10000. Loss: 5.308\n",
      "Gen #600 out of 10000. Loss: 5.269\n",
      "Gen #650 out of 10000. Loss: 5.234\n",
      "Gen #700 out of 10000. Loss: 5.18\n",
      "Gen #750 out of 10000. Loss: 5.351\n",
      "Gen #800 out of 10000. Loss: 5.521\n",
      "Gen #850 out of 10000. Loss: 5.276\n",
      "Gen #900 out of 10000. Loss: 4.909\n",
      "Gen #950 out of 10000. Loss: 5.045\n",
      "Gen #1000 out of 10000. Loss: 5.31\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['das', 'ist', 'ist', 'ist', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'ist', 'die', 'die', 'der', 'der', 'der', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['das', 'ist', 'ist', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #1050 out of 10000. Loss: 5.326\n",
      "Gen #1100 out of 10000. Loss: 4.878\n",
      "Gen #1150 out of 10000. Loss: 4.904\n",
      "Gen #1200 out of 10000. Loss: 4.976\n",
      "Gen #1250 out of 10000. Loss: 4.717\n",
      "Gen #1300 out of 10000. Loss: 4.854\n",
      "Gen #1350 out of 10000. Loss: 5.082\n",
      "Gen #1400 out of 10000. Loss: 4.977\n",
      "Gen #1450 out of 10000. Loss: 4.844\n",
      "Gen #1500 out of 10000. Loss: 4.96\n",
      "English: hello where is my computer\n",
      "German: ['es', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'zu']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'ist', 'die', 'die', 'die', 'die', 'die', 'die', 'die', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['es', 'ist', 'ist', 'ist', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #1550 out of 10000. Loss: 4.453\n",
      "Gen #1600 out of 10000. Loss: 5.135\n",
      "Gen #1650 out of 10000. Loss: 5.058\n",
      "Gen #1700 out of 10000. Loss: 4.691\n",
      "Gen #1750 out of 10000. Loss: 4.159\n",
      "Gen #1800 out of 10000. Loss: 4.77\n",
      "Gen #1850 out of 10000. Loss: 4.661\n",
      "Gen #1900 out of 10000. Loss: 4.655\n",
      "Gen #1950 out of 10000. Loss: 4.978\n",
      "Gen #2000 out of 10000. Loss: 4.655\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['das', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'ist', 'ist', 'der', 'der', 'der', 'der', 'der', 'der', 'der', 'der', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['es', 'ist', 'es', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #2050 out of 10000. Loss: 4.499\n",
      "Gen #2100 out of 10000. Loss: 4.72\n",
      "Gen #2150 out of 10000. Loss: 4.111\n",
      "Gen #2200 out of 10000. Loss: 4.544\n",
      "Gen #2250 out of 10000. Loss: 3.87\n",
      "Gen #2300 out of 10000. Loss: 4.362\n",
      "Gen #2350 out of 10000. Loss: 4.252\n",
      "Gen #2400 out of 10000. Loss: 4.792\n",
      "Gen #2450 out of 10000. Loss: 4.404\n",
      "Gen #2500 out of 10000. Loss: 4.112\n",
      "English: hello where is my computer\n",
      "German: ['mein', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['die', 'ist', 'ist', 'die', 'die', 'die', 'die', 'die', 'sie', 'sie', 'sie', 'sie', 'sie', 'sie', 'sie', 'sie', 'sie']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['es', 'ist', 'es', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #2550 out of 10000. Loss: 4.865\n",
      "Gen #2600 out of 10000. Loss: 3.89\n",
      "Gen #2650 out of 10000. Loss: 3.872\n",
      "Gen #2700 out of 10000. Loss: 4.871\n",
      "Gen #2750 out of 10000. Loss: 4.521\n",
      "Gen #2800 out of 10000. Loss: 4.469\n",
      "Gen #2850 out of 10000. Loss: 4.437\n",
      "Gen #2900 out of 10000. Loss: 4.789\n",
      "Gen #2950 out of 10000. Loss: 4.644\n",
      "Gen #3000 out of 10000. Loss: 4.766\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['meine', 'ist', 'ist', 'meine', 'meine', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['die', 'der', 'hat', 'die', 'der', 'der', 'der', 'der', 'die', 'die', 'die', 'die', 'die', 'die', 'die', 'die', 'die']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['es', 'es', 'es', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #3050 out of 10000. Loss: 4.368\n",
      "Gen #3100 out of 10000. Loss: 4.578\n",
      "Gen #3150 out of 10000. Loss: 4.71\n",
      "Gen #3200 out of 10000. Loss: 3.402\n",
      "Gen #3250 out of 10000. Loss: 3.923\n",
      "Gen #3300 out of 10000. Loss: 4.329\n",
      "Gen #3350 out of 10000. Loss: 4.034\n",
      "Gen #3400 out of 10000. Loss: 4.64\n",
      "Gen #3450 out of 10000. Loss: 4.357\n",
      "Gen #3500 out of 10000. Loss: 4.76\n",
      "English: hello where is my computer\n",
      "German: ['mein', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'der', 'hat', 'sich', 'sich', 'die', 'die', 'die', 'die', 'die', 'die', 'war', 'war', 'war', 'war', 'war', 'war']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['es', 'es', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #3550 out of 10000. Loss: 3.661\n",
      "Gen #3600 out of 10000. Loss: 3.399\n",
      "Gen #3650 out of 10000. Loss: 3.954\n",
      "Gen #3700 out of 10000. Loss: 4.573\n",
      "Gen #3750 out of 10000. Loss: 4.395\n",
      "Gen #3800 out of 10000. Loss: 4.096\n",
      "Gen #3850 out of 10000. Loss: 4.525\n",
      "Gen #3900 out of 10000. Loss: 4.456\n",
      "Gen #3950 out of 10000. Loss: 4.485\n",
      "Gen #4000 out of 10000. Loss: 3.908\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['meine', 'ist', 'meine', 'meine', 'meine', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'wurde', 'die', 'die', 'die', 'die', 'die', 'die', 'die', 'die', 'der', 'war', 'war', 'war', 'war', 'war']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['es', 'es', 'es', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #4050 out of 10000. Loss: 3.526\n",
      "Gen #4100 out of 10000. Loss: 3.986\n",
      "Gen #4150 out of 10000. Loss: 3.713\n",
      "Gen #4200 out of 10000. Loss: 3.752\n",
      "Gen #4250 out of 10000. Loss: 3.774\n",
      "Gen #4300 out of 10000. Loss: 3.7\n",
      "Gen #4350 out of 10000. Loss: 4.049\n",
      "Gen #4400 out of 10000. Loss: 3.515\n",
      "Gen #4450 out of 10000. Loss: 3.918\n",
      "Gen #4500 out of 10000. Loss: 4.466\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['die', 'mann', 'hat', 'die', 'die', 'die', 'die', 'die', 'die', 'hat', 'hat', 'hat', 'hat', 'hat', 'hat', 'hat', 'hat']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['ist', 'es', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #4550 out of 10000. Loss: 3.575\n",
      "Gen #4600 out of 10000. Loss: 4.065\n",
      "Gen #4650 out of 10000. Loss: 3.231\n",
      "Gen #4700 out of 10000. Loss: 3.409\n",
      "Gen #4750 out of 10000. Loss: 3.144\n",
      "Gen #4800 out of 10000. Loss: 4.42\n",
      "Gen #4850 out of 10000. Loss: 3.69\n",
      "Gen #4900 out of 10000. Loss: 4.285\n",
      "Gen #4950 out of 10000. Loss: 4.393\n",
      "Gen #5000 out of 10000. Loss: 4.216\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'meine', 'meine', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'die', 'die', 'die', 'die', 'die', 'die', 'die', 'die', 'war', 'war', 'hat', 'hat', 'hat', 'hat']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['ist', 'es', 'morgen', 'morgen', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #5050 out of 10000. Loss: 3.256\n",
      "Gen #5100 out of 10000. Loss: 3.782\n",
      "Gen #5150 out of 10000. Loss: 3.63\n",
      "Gen #5200 out of 10000. Loss: 3.249\n",
      "Gen #5250 out of 10000. Loss: 3.444\n",
      "Gen #5300 out of 10000. Loss: 4.069\n",
      "Gen #5350 out of 10000. Loss: 3.32\n",
      "Gen #5400 out of 10000. Loss: 4.405\n",
      "Gen #5450 out of 10000. Loss: 3.054\n",
      "Gen #5500 out of 10000. Loss: 3.439\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'meine', 'meine', 'meine', 'meine', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'die', 'die', 'die', 'die', 'die', 'den', 'den', 'den', 'zu', 'hatte', 'hat', 'hat', 'hat', 'hat']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['ist', 'es', 'morgen', 'morgen', 'morgen', 'morgen', 'zu', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #5550 out of 10000. Loss: 3.686\n",
      "Gen #5600 out of 10000. Loss: 3.612\n",
      "Gen #5650 out of 10000. Loss: 2.975\n",
      "Gen #5700 out of 10000. Loss: 4.207\n",
      "Gen #5750 out of 10000. Loss: 4.103\n",
      "Gen #5800 out of 10000. Loss: 3.184\n",
      "Gen #5850 out of 10000. Loss: 4.118\n",
      "Gen #5900 out of 10000. Loss: 3.51\n",
      "Gen #5950 out of 10000. Loss: 3.406\n",
      "Gen #6000 out of 10000. Loss: 4.096\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'meine', 'meine', 'mein', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'die', 'die', 'die', 'die', 'die', 'die', 'die', 'hatte', 'hatte', 'hatte', 'hatte', 'hatte', 'hatte', 'hatte']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['ist', 'es', 'morgen', 'morgen', 'morgen', 'morgen', 'morgen', 'zu', 'zu', 'zu', 'zu', 'zu']\n",
      "Gen #6050 out of 10000. Loss: 3.151\n",
      "Gen #6100 out of 10000. Loss: 3.761\n",
      "Gen #6150 out of 10000. Loss: 3.79\n",
      "Gen #6200 out of 10000. Loss: 3.692\n",
      "Gen #6250 out of 10000. Loss: 4.191\n",
      "Gen #6300 out of 10000. Loss: 3.054\n",
      "Gen #6350 out of 10000. Loss: 3.479\n",
      "Gen #6400 out of 10000. Loss: 3.183\n",
      "Gen #6450 out of 10000. Loss: 3.441\n",
      "Gen #6500 out of 10000. Loss: 4.02\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'meine', 'meine', 'meine', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'den', 'den', 'den', 'den', 'den', 'den', 'den', 'den', 'der', 'war', 'war', 'war', 'war', 'war']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['wird', 'es', 'morgen', 'morgen', 'morgen', 'morgen', 'morgen', 'morgen', 'gehen', 'gehen', 'gehen', 'gehen']\n",
      "Gen #6550 out of 10000. Loss: 3.493\n",
      "Gen #6600 out of 10000. Loss: 3.04\n",
      "Gen #6650 out of 10000. Loss: 3.078\n",
      "Gen #6700 out of 10000. Loss: 3.51\n",
      "Gen #6750 out of 10000. Loss: 3.814\n",
      "Gen #6800 out of 10000. Loss: 3.468\n",
      "Gen #6850 out of 10000. Loss: 3.268\n",
      "Gen #6900 out of 10000. Loss: 4.03\n",
      "Gen #6950 out of 10000. Loss: 3.029\n",
      "Gen #7000 out of 10000. Loss: 3.59\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'meine', 'meine', 'meine', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'sich', 'die', 'die', 'die', 'mann', 'mann', 'mann', 'mann', 'hat', 'hat', 'hat', 'hat', 'hat', 'hat']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['wird', 'es', 'morgen', 'morgen', 'morgen', 'morgen', 'gehen', 'gehen', 'gehen', 'werden', 'werden', 'werden']\n",
      "Gen #7050 out of 10000. Loss: 2.631\n",
      "Gen #7100 out of 10000. Loss: 2.974\n",
      "Gen #7150 out of 10000. Loss: 3.306\n",
      "Gen #7200 out of 10000. Loss: 3.555\n",
      "Gen #7250 out of 10000. Loss: 3.943\n",
      "Gen #7300 out of 10000. Loss: 3.932\n",
      "Gen #7350 out of 10000. Loss: 3.583\n",
      "Gen #7400 out of 10000. Loss: 3.593\n",
      "Gen #7450 out of 10000. Loss: 2.833\n",
      "Gen #7500 out of 10000. Loss: 3.738\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'mein', 'mein', 'mein', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'sich', 'die', 'die', 'die', 'die', 'hund', 'hund', 'hund', 'hund', 'hund', 'wurde', 'wurde', 'wurde', 'wurde']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['wird', 'es', 'morgen', 'morgen', 'morgen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen']\n",
      "Gen #7550 out of 10000. Loss: 3.515\n",
      "Gen #7600 out of 10000. Loss: 2.954\n",
      "Gen #7650 out of 10000. Loss: 3.933\n",
      "Gen #7700 out of 10000. Loss: 3.252\n",
      "Gen #7750 out of 10000. Loss: 2.993\n",
      "Gen #7800 out of 10000. Loss: 3.597\n",
      "Gen #7850 out of 10000. Loss: 3.499\n",
      "Gen #7900 out of 10000. Loss: 4.009\n",
      "Gen #7950 out of 10000. Loss: 3.716\n",
      "Gen #8000 out of 10000. Loss: 2.477\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'mein', 'mein', 'mein', 'mein', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'sich', 'die', 'die', 'die', 'hund', 'hund', 'hund', 'hund', 'wurde', 'wurde', 'wurde', 'wurde', 'wurde', 'wurde']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['ist', 'es', 'morgen', 'morgen', 'morgen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen']\n",
      "Gen #8050 out of 10000. Loss: 3.264\n",
      "Gen #8100 out of 10000. Loss: 2.105\n",
      "Gen #8150 out of 10000. Loss: 3.692\n",
      "Gen #8200 out of 10000. Loss: 2.701\n",
      "Gen #8250 out of 10000. Loss: 3.574\n",
      "Gen #8300 out of 10000. Loss: 2.88\n",
      "Gen #8350 out of 10000. Loss: 3.554\n",
      "Gen #8400 out of 10000. Loss: 3.779\n",
      "Gen #8450 out of 10000. Loss: 2.547\n",
      "Gen #8500 out of 10000. Loss: 3.565\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'meine', 'mein', 'mein', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'sich', 'sich', 'die', 'die', 'die', 'hund', 'hund', 'hund', 'hund', 'hund', 'hat', 'hat', 'hat', 'hat']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['wird', 'es', 'morgen', 'morgen', 'morgen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen', 'muss']\n",
      "Gen #8550 out of 10000. Loss: 2.268\n",
      "Gen #8600 out of 10000. Loss: 2.799\n",
      "Gen #8650 out of 10000. Loss: 2.679\n",
      "Gen #8700 out of 10000. Loss: 3.278\n",
      "Gen #8750 out of 10000. Loss: 3.373\n",
      "Gen #8800 out of 10000. Loss: 2.632\n",
      "Gen #8850 out of 10000. Loss: 3.302\n",
      "Gen #8900 out of 10000. Loss: 3.145\n",
      "Gen #8950 out of 10000. Loss: 2.752\n",
      "Gen #9000 out of 10000. Loss: 3.807\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'meine', 'meine', 'mein', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'sich', 'sich', 'die', 'die', 'die', 'die', 'die', 'hund', 'hund', 'hund', 'hund', 'hat', 'hat', 'hat']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['wird', 'es', 'morgen', 'morgen', 'morgen', 'morgen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen']\n",
      "Gen #9050 out of 10000. Loss: 2.773\n",
      "Gen #9100 out of 10000. Loss: 2.477\n",
      "Gen #9150 out of 10000. Loss: 2.899\n",
      "Gen #9200 out of 10000. Loss: 2.985\n",
      "Gen #9250 out of 10000. Loss: 2.741\n",
      "Gen #9300 out of 10000. Loss: 3.437\n",
      "Gen #9350 out of 10000. Loss: 2.529\n",
      "Gen #9400 out of 10000. Loss: 3.025\n",
      "Gen #9450 out of 10000. Loss: 3.395\n",
      "Gen #9500 out of 10000. Loss: 2.775\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'ist', 'meine', 'meine', 'ist', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'sich', 'ber', 'die', 'die', 'hund', 'hund', 'hund', 'hund', 'hund', 'hund', 'hund', 'hund', 'hund', 'hund']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['wird', 'es', 'morgen', 'morgen', 'morgen', 'morgen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen', 'muss']\n",
      "Gen #9550 out of 10000. Loss: 2.758\n",
      "Gen #9600 out of 10000. Loss: 2.622\n",
      "Gen #9650 out of 10000. Loss: 3.466\n",
      "Gen #9700 out of 10000. Loss: 2.492\n",
      "Gen #9750 out of 10000. Loss: 2.718\n",
      "Gen #9800 out of 10000. Loss: 3.33\n",
      "Gen #9850 out of 10000. Loss: 2.351\n",
      "Gen #9900 out of 10000. Loss: 2.164\n",
      "Gen #9950 out of 10000. Loss: 3.074\n",
      "Gen #10000 out of 10000. Loss: 2.125\n",
      "Saving model to temp/seq2seq_model.\n",
      "English: hello where is my computer\n",
      "German: ['wo', 'wo', 'mein', 'mein', 'mein', 'ist', 'ist', 'ist', 'ist', 'ist']\n",
      "English: the quick brown fox jumped over the lazy dog\n",
      "German: ['der', 'mann', 'hat', 'sich', 'ber', 'ber', 'den', 'hund', 'hund', 'hund', 'hund', 'hund', 'hund', 'hat', 'hat', 'hat', 'hat']\n",
      "English: is it going to rain tomorrow\n",
      "German: ['wird', 'es', 'morgen', 'morgen', 'morgen', 'regnen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen', 'kommen']\n"
     ]
    }
   ],
   "source": [
    "# Start training\n",
    "train_loss = []\n",
    "for i in range(generations):\n",
    "    rand_bucket_ix = np.random.choice(len(bucketed_data))\n",
    "    \n",
    "    model_outputs = translate_model.get_batch(bucketed_data, rand_bucket_ix)\n",
    "    encoder_inputs, decoder_inputs, target_weights = model_outputs\n",
    "    \n",
    "    # Get the (gradient norm, loss, and outputs)\n",
    "    _, step_loss, _ = translate_model.step(sess, encoder_inputs, decoder_inputs,\n",
    "                                           target_weights, rand_bucket_ix, False)\n",
    "    \n",
    "    # Output status\n",
    "    if (i+1) % output_every == 0:\n",
    "        train_loss.append(step_loss)\n",
    "        print('Gen #{} out of {}. Loss: {:.4}'.format(i+1, generations, step_loss))\n",
    "    \n",
    "    # Check if we should decay the learning rate\n",
    "    if (i+1) % lr_decay_every == 0:\n",
    "        sess.run(translate_model.learning_rate_decay_op)\n",
    "        \n",
    "    # Save model\n",
    "    if (i+1) % save_every == 0:\n",
    "        print('Saving model to {}.'.format(full_model_dir))\n",
    "        model_save_path = os.path.join(full_model_dir, \"eng_ger_translation.ckpt\")\n",
    "        translate_model.saver.save(sess, model_save_path, global_step=i)\n",
    "        \n",
    "    # Eval on test set\n",
    "    if (i+1) % eval_every == 0:\n",
    "        for ix, sentence in enumerate(test_data):\n",
    "            # Find which bucket sentence goes in\n",
    "            bucket_id = next(index for index, val in enumerate(x_maxs) if val>=len(sentence))\n",
    "            # Get RNN model outputs\n",
    "            encoder_inputs, decoder_inputs, target_weights = test_model.get_batch(\n",
    "                {bucket_id: [(sentence, [])]}, bucket_id)\n",
    "            # Get logits\n",
    "            _, test_loss, output_logits = test_model.step(sess, encoder_inputs, decoder_inputs,\n",
    "                                                           target_weights, bucket_id, True)\n",
    "            ix_output = [int(np.argmax(logit, axis=1)) for logit in output_logits]\n",
    "            # If there is a 0 symbol in outputs end the output there.\n",
    "            ix_output = ix_output[0:[ix for ix, x in enumerate(ix_output+[0]) if x==0][0]]\n",
    "            # Get german words from indices\n",
    "            test_german = [ger_ix2vocab[x] for x in ix_output]\n",
    "            print('English: {}'.format(test_english[ix]))\n",
    "            print('German: {}'.format(test_german))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Here is matplotib code to plot the training loss over the 10000 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEZCAYAAACTsIJzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4XNWZ/7+vumY06nVkybaqq1xxw4CoNi6QEFrWtIRk\ngU2AkGVhYbOYHsomlJD8WBICS4kxbAp1YxywwNjGFWMZ9Wqra9Rm1Nv5/TFzru+drjLS2Lyf55mH\nmdvOuVfmfO/3fU8hIQQYhmEYRhIw3RVgGIZh/AsWBoZhGEYDCwPDMAyjgYWBYRiG0cDCwDAMw2hg\nYWAYhmE0sDAwDMMwGlgYzkCIaC0R7SGiTiIyEdFuIlo23fWaCoholIgyJnD+zURUTERdRNRIRO8T\nkX4y6+iPENF5RHRyuuvB+AdB010BZnIhIgOA9wHcAuAdACEAzgEwMJ31mkLGPWKTiM4D8BiAS4QQ\nx4goGsDmSauZ/8OjXRkA7BjORHIACCHE28LKgBDiH0KI4/IAIvohERURURsR/R8Rpav2XWx7Y+4g\not8QUQER/dC2bysRva46dqbtDT3A9juSiP5ARA1EdJKIHiEisu270eZcniaidiKqJKL1qmvFENEf\niajeVq+/qPZtIqKvbHX6gogWOrtxIvoMAAE4RkRmIrrKtv3HRFRuc09/I6IUF89uOYC9QohjsD7E\nTiHE60KIHtt1Qojov4io1uYmfkdEoary/81273VE9AO1eyGiXfI5qp+H6vccIvrYdu/Fsu62fa8Q\n0QtE9IHtvvYR0WzV/vmqcxuJ6N9t24mI/p2IKoiolYjesondmLD9XV8johYiqiai/1Dty7T9G+m0\n7d+m2vcMETXb9h0lonljLZuZHlgYzjzKAIwQ0atEtN6+ISCi7wD4dwDfAZAAYDeAbbZ98QD+F8D9\nAOIBVAJYY3d9+7dK9e/XAAwCyACwBMDFAH6k2r8CQDGAOABPA3hZte8NAOEA5gJIBPCMrU5Lbcf9\nGEAsgP8G8B4RBdvfuBDiPNvXhUKISCHEO0R0AYDHAVwJIAXACQBv2Z9rYz+AdUT0IBGtIaIQu/1P\nAcgCkGf7byqAB2z1XA/g5wAuBJAN4CJ4fgMXtnN1AD62PYN4AN8H8Dsimqs69loAWwFEw/p3ecx2\nbgSAnQA+st1fFoBPbOfcCeAyWB2jEUAHgN95qJMzXgBgADALQD6AG4joB7Z9jwDYIYSIBjADwG9s\n9boEwFoAWbZ91wBoG0fZzHQghODPGfYBkAvgj7A2goMA3gWQYNv3EYAfqI4NANADIA3A9bC+Mauv\ndRLAD23ftwJ4TbVvJoAR2zWSAPQDCFXtvxbAp7bvNwIoU+0LBzAKqwgkAxgGEOnkXn4H4CG7bSUA\nznFx76MAMlS//wDgCdVvve2ZpLs4f53tebUDMAP4FQCy7esGMFt17GoAVbbvLwN4XLUv2/ZsMmy/\nd8nnqHoen9u+Xw3gM7t6vAjgP23fXwHwkmrfpQCKbN+/D+Cwi3spAnC+6neK7d4DnBx7HoATTrYH\n2P6uuapt/6z6u/6Pra6pduedb/s7rZTPjz+nz4cdwxmIEKJUCPFDIUQ6gAWwvi0+a9s9E8BztnBO\nO6xvcQLWt18jrEKgxtuEZDqAYACNtmt3wNpgxKuOaVLVsc/2NQJWUWoXQpidXHcmgH+V9bVdd4at\nrt5gBFCrKrcH1ntOdXawEGKHEOJyIUQsgMsB3ATgR0SUAEAH4LDq2f0frO5HlqN+VrWwhrW8YSaA\nVXb3+E+wiq2kSfW9F9bnBlifRaWb6/5VVd8iAEN21/VEPKx/1xOqbbU49fzugVU8DhBRoXQSQohd\nsDqN3wJoIqIXbe6GOQ1gYTjDEUKUAXgVVoEArI3XLUKIWNsnRggRIYT4EkAjrA28mjTV9x5YG0eJ\nOlZ/EtY3yzjVdaOFEHleVPMkgFgiinSx7zEn9d3uxXUBoAHWBhIAQNYeRnEA6j2daGvcPoX12Zlg\nbZDnq+oSLYSIsh3eCO2zmgltKMn+2SXb3WOB3T1GCiF+6sX9nYQ1fOSMEwAutbuuXgjR6MV1JSZY\nxWSmattM2J6fEKJZCPHPQohUALfCGgLLsO17QQixHMB8WF3sv42hXGYaYWE4wyCiXCL6ORGl2n6n\nwRpu2Gc75EUA98tEIBFFEdGVtn0fAphHRN8hokAiuhPat8ujAM4lojQiioI1VwEAEEI0wRonf4aI\nDLbEZwYRneupzrZz/w/WRiWaiIKI6Bzb7t8DuJWIVtjqqyeiDeS6C2kTrDkOyZ8A/ICI8myJ4scB\nfCmEOGF/IhFdRkTXyLyMrczzAOwTQghbXZ61uQcQUaotlg4AbwO4iYjm2nIGD9hd/iiAK4gonIiy\nANys2vcBgBwius5278FEtJyIcj09O9u5SUR0B1mT4xHyWcGaj3mcbJ0LiCiBiC5zcy0iolD1Rwgx\naru3x2zXngngLgCv2064Uv5bA9AJayhvxFb/FUQUBKAP1peGES/uh/EDWBjOPCywxnX3E5EFwF4A\nxwDcDQBCiL8BeALAW0TUadu33ravDcBVAJ6E9U0xE8AeeWEhxD8AbLedcxDWbrFqboC1e2wRrDH6\nd6B9M7ZH/UZ9Pax5hhIAzbAmTiGEOAxr4vkFWzikDNb4vCseBPCaLXxypRDiUwD/CeAvsL7lzoY1\n9+GMDltZZUTUBWsy/UkhhExW3wugAsCXtmf3May9wCCE+Dus4bpPbXX8xO7az8D65t0Ea87gDeUh\nCNEN4BJbvRpsnycAhMIDtnMvhjXJ3GQrO9+2+zlY8yUf2+5nL6wdAFxhhNUV9cLamPfa3v7vsG2r\nAvA5gDeEEK/YzjkL1n9rZgB/A3CHEKIWQCSsQtoOoBrWf0//5el+GP9AJtV8V4D1rVP2TPm9EOJ5\nnxbITCpEtAvA60KIP053XU43iGgU1l45VdNdF4YZCz51DEQ0H1bLvBzAYgCbiSjTl2UyDMMwE8PX\noaS5sMZzB4QQIwA+A/BdH5fJTC48Gnb88LNjTkt8PSXGcQCPElEMrFMybIA1Ns2cJgghLpjuOpyu\nCCECp7sODDMefCoMQogSInoSwD9gTYoehTXByDAMw/gpPk8+awojegzASSHEi3bb2XIzDMOMESGE\nt4Mox4TPu6uq+nynw5pf2ObsuOkeAu4vn61bt057Hfzhw8+BnwU/C/cfXzIV027/mYhiYe3D/S9C\niK4pKJNhGIYZJz4XBiGEx5GvDMMwjP/AI5/9jPz8/Omugl/Az+EU/CxOwc9iapjS5LPLShAJf6gH\nwzDM6QIRQZyuyWeGYRjm9IKFgWEYhtHAwsAwDMNoYGFgGIZhNLAwMAzDMBpYGBiGYRgNLAwMwzCM\nBhYGhmEYRgMLA8MwDKOBhYFhGIbRwMLAMAzDaGBhYBiGYTSwMDAMwzAaWBgYhmEYDSwMDMMwjAYW\nBoZhGEYDCwPDMAyjgYWBYRiG0cDCwDAMw2hgYWAYhmE0sDAwDMMwGnwuDER0FxEdJ6JjRPQmEYX4\nukyGYRhm/PhUGIjICOB2AEuFEHkAggBc68syGYZhmIkRNAVlBALQE9EoAB2Ahikok2EYhhknPnUM\nQogGAL8CcAJAPYBOIcQ/fFkmwzAMMzF86hiIKBrA5QBmAugC8L9E9E9CiD/ZH/vggw8q3/Pz85Gf\nn+/LqjEMw5xWFBQUoKCgYErKIiGE7y5OdCWAdUKIH9t+Xw9gpRDip3bHCV/Wg2EY5kyDiCCEIF9c\n29e9kk4AWEVEYUREAC4EUOzjMhmGYZgJ4OscwwEA/wvgKwBfAyAAL/myTIZhGGZi+DSU5HUlOJTE\nMAwzJk7nUBLDMAxzmsHCwDAMw2hgYWAYhmE0sDAwDMMwGlgYGIZhGA0sDAzDMIwGFgaGYRhGAwsD\nwzAMo4GFgWEYhtHAwsAwDMNoYGFgGIZhNLAwMAzDMBpYGBiGYRgNLAwMwzCMBhYGhmEYRgMLA8Mw\nDKOBhYFhGIbRwMLAMAzDaGBhYBiGYTSwMDAMwzAaWBgYhmEYDSwMDMMwjAYWBoZhGEaDT4WBiHKI\n6CsiOmL7bxcR3eHLMhmGYZiJQUKIqSmIKABAHYCVQoiTdvvEVNWDYRjmTICIIIQgX1x7KkNJFwGo\ntBcFhmEYxr+YSmG4BsC2KSyPYRiGGQdBU1EIEQUDuAzAv7s6ZuvWrSCyuqL8/Hzk5+dPRdUYhmFO\nCwoKClBQUDAlZU1JjoGILgPwL0KI9S72i/7+foSGhvq8LgzDMGcCZ0KO4fvwEEYaGhqaoqowDMMw\n7vC5MBBROKyJ57+4O25wcNDXVWEYhmG8wOc5BiFEH4AET8exY2AYhvEP/GbkMzsGhmEY/4CFgWEY\nhtHgN8LAoSSGYRj/wG+EgR0DwzCMf+A3wsCOgWEYxj/wG2Fgx8AwDOMfsDAwDMMwGvxGGDiUxDAM\n4x/4jTCwY2AYhvEP/EYY2DEwDMP4B34jDOwYGIZh/AO/EQZ2DAzDMP6B3wgDOwaGYRj/gIWBYRiG\n0eA3wsChJIZhGP/Ab4SBHQPDMIx/4DfCwI6BYRjGP/AbYWDHwDAM4x+wMDAMwzAa/EYYOJTEMAzj\nH/iNMLBjYBiG8Q/8RhjYMTAMw/gHfiMM7BgYhmH8A58LAxFFEdE7RFRMRN8Q0Upnx7FjYBiG8Q+C\npqCM5wB8JIS4ioiCAOicHcSOgWEYxj/wqTAQkQHAOUKImwBACDEMwOzsWBYGhmEY/8DXoaQMACYi\neoWIjhDRS0QU7uxADiUxDMP4B74OJQUBWArgJ0KIQ0T0LIB/B7DV/sDjx4/jwQcfBADk5+cjPz/f\nx1VjGIY5fSgoKEBBQcGUlEVCCN9dnCgJwD4hRIbt91oA9wohNtsdJy655BLs2LHDZ3VhGIY5kyAi\nCCHIF9f2aShJCNEM4CQR5dg2XQigyNmxnZ2dvqwKwzAM4yVTMY7hDgBvEtFRAIsAPO7soKKiInR1\ndQEA3nrrLU5GMwzDTBM+DSV5XQkicfHFF+O2227DwoULkZ2djU8//RTnn3/+dFeNYRjGLzltQ0lj\nYf369dixYwfeeOMNhIeH45NPPpnuKjEMw3wr8RvHcPz4cWzcuBGBgYG45ZZb8Le//Q179+6d7qox\nDMP4Jb50DH4jDKOjo0hLS0NkZCQOHz6MhIQENDQ0IDIycrqrxzAM43dMeyiJiDKJKNT2PZ+I7iCi\n6MmsCBHhqquuwq233orw8HCsWLECu3fvnswiGIZhGC/wyjHYehQtBzALwEcA3gUwXwixYVIqQSSE\nEBBCgMgqgI8++ija29vx61//ejKKYBiGOaOYdscAYNQ2z9F3AfxGCPFvAFImuzJSFABg48aN2LZt\nG2praye7GIZhGMYN3grDEBF9H8CNAD6wbQv2TZWsLFmyBPfeey8uvfRSdHR0+LIohmEYRoW3wvAD\nAKsBPCaEqCai2QDe8F21rPzsZz/DRRddhJtvvhnqkNfzzz/P02cwDMP4CK+EQQhRJIS4QwixjYhi\nABiEEE/4uG4AgKeffhqVlZV47bXXIITA448/jnvuuQdvvvnmVBTPMAzzrcPb5HMBgMtgnS31MIAW\nAHuEED+flErYks+uOHbsGM477zwEBAQgJSUFv/rVr3DnnXeipKRkMopnGIY57Zj2cQxE9JUQYgkR\n/QhAmhBiKxEdE0LkTUolPAgDANTW1iIsLAyJiYkYGRlBdHQ06urqEB3tudfshRdeiN/+9reYM2fO\nZFSXYRhm2vGHXklBRJQC4GqcSj5PKTNnzkRSUhKICEFBQVi6dCkOHTrk8byKigp8+umn+Oyzz6ag\nlgzDMKc/3grDwwB2AKgUQhwkogwA5b6rlmdWrFiBAwcOeDzuvffeQ2RkJPbv3z8FtWIYhjn98Tb5\n/I4QIk8IcZvtd5UQ4nu+rZp7vBWGd999F3fffTcLA8MwjJd4m2OYAeA3AM4GIAB8AeBOIUTdpFTC\nixyDPbW1tVi1ahUaGho0A+PUmEwmZGZmoq6uDikpKTz3EsMwZwz+kGN4BcB7AIwAUgG8b9s2baSn\npwOA2/mUPvzwQ1x44YUwGAxYvHixVzkJhmGYbzveCkOCEOIVIcSw7fMqgAQf1ssjRIRXXnkFV155\npcsFst99911cfvnlAICVK1c6DSd9/PHHuOWWW7B9+3ZfVpdhGOa0wVthMBHRdUQUaPtcB6DNlxXz\nhvXr12P79u24+uqr8dvf/lYzOrqvrw+ffPIJNm7cCMCak7AXhm+++QZbtmxBW1sb3nvvvSmtO8Mw\njL/irTD8ENauqk0AGgFcCes0GdPO+eefj7179+IPf/gDrrjiCjQ1NQEAPvnkEyxatAjx8fEAgLPO\nOgtHjhzRnHvw4EFccskluP3223HixIkprzvDMIw/4m2vpBNCiMuEEAlCiEQhxHcAXOHjunlNVlYW\nvvzyS8ybNw+LFi3Cvn378N577ylhJMA6DqK1tRV9fX3KtsOHD2PZsmVIT0/HyZMnp6PqDMMwfkfQ\nBM79OYBnJ6siEyU0NBSPPfYYzjnnHHznO9/B8PCwJnQUGBiIWbNmobKyEgsWLABgFYbvfe97SE1N\nRWNjI0ZGRhAYGDhdt8AwDOMXeBtKcoZPuklNlPXr1+Pll1/Gueeei6ysLM2+rKwslJdbx+UNDw/j\n2LFjWLJkCUJCQhAfH4/GxsbpqDLDMIxfMRFhmP7Fol2wadMm/PWvf3XYnp2djYqKCgBAaWkpjEYj\noqKiAFi7v040zzAyMoKtW7diYGBgQtdhGIaZTtwKAxFZiMjs5GOBdUyDR4iohoi+JqKviMjzUGUf\nkp2drTiGw4cPY+nSpcq+yRAGk8mEhx9+GFu3bp3QdRiGYaYTt8IghDAIISKdfAxCCG/zE6MA8oUQ\nS4QQKyZe5fGjDiXJxLPEG2HYvn07brvtNpf729raYDQa8T//8z/Yu3fv5FSaYRhmiplIKMlbaIrK\n8Yg6lLR///4xC8P777+P//7v/0ZxcbHT/SaTCbNnz8bDDz+MX//615NXcYZhmClkKhpsAWAHER0k\noh9PQXkuSUtLg8lkQmFhIaqrq7F27VplnzfCsGfPHlx//fV45JFHnO43mUyIj4/H8uXLUVZWNql1\nZxiGmSom0l3VW9YIIZqIKAHATiIqFkJ8YX/Qgw8+qHzPz89Hfn7+pFdEdln9j//4D1x33XUICQlR\n9tkLw5EjR2CxWHDeeecBABoaGmCxWPCb3/wGWVlZKC0tRW5uLkZHRzEyMoLg4GC0tbUhPj4eWVlZ\nqKyshBDC5QR/gHWtiC+//BLXXXfdpN8rwzBnFgUFBS6n/5l0hBBT9gGwFcDPnWwXU8XmzZsFAPHN\nN99otre2toqYmBghhBAtLS3CaDSKjRs3KvvffvttsXnzZiGEEPfff7/42c9+JoQQ4sknnxQ//vGP\nhRBCPP744+Kee+4RQgiRlJQk6urq3Nbl+eefF8uWLZucG2MY5luFrd30SVvt01ASEemIKML2XQ/g\nEgDHfVmmJ7Kzs7Fq1SrMmzdPsz0uLg79/f0wmUy48cYbcf755+PQoUPK/Et79uzBmjVrAAA333wz\n3njjDZjNZjzzzDOoqqoCcCqUJMuR+QxXFBUVobi4GKOjo5N9mwzDMOPG1zmGJABfENFXAL4E8L4Q\n4mMfl+mWG2+8Ec8884zDdiJCeno65syZA71ej1deeQVCCNTX1wOwCsPZZ58NAMjIyMDixYtxzTXX\nQKfTKcfIUBKg7QHliuLiYvT29vp0Og4hBBoaGnx2fYZhzjx8KgxCiGohxGJh7aq6UAjxhC/L84a8\nvDysWrXK6b5HH30UH374Id555x0EBwdj+fLlOHToECwWC4qKirB8+XLl2B//+Mf4+9//jieeeEJp\neO0dgydhKCoqwvz581FUVDQp9/b1119j/fr1mm2HDx/GJZdcMinXZxjm24FfdCP1F6688kqsXLlS\n+S2F4Z133sFFF12E8PBwZd93vvMdPPvss7jyyisxNDSE7u5umEwmxMXFAbA6BnehJJPJhMHBQVxw\nwQWTJgwnTpxAXZ12Ub329naUlZVheHh4UspgGObMh4XBDVIYXn75Zdx8882afSEhIbjzzjtBRDAa\njWhoaNCEkpw5BpmvAKxhpHnz5mH+/Pkux0W4o7a2Fk8++aRmW3t7O8xms2abxWLB0NAQamtrHa7x\n2WefTV0vB4ZhThtYGNywbNkyfPbZZ6isrMSll17q8rjU1FTU19drQknqLquAdZnRTZs2KecUFRVh\n7ty5mDt37rgcw6FDh/CHP/xBs82ZMMjfpaWlDtd4/fXXlek7SkpKcOWVV465Hmq+/PJLp3NUMQxz\nesHC4Aaj0YjY2FjccMMNCA4OdnvciRMnYLFYEB0dDQAwGAwwGAxK/qGqqgofffQRvvzySwBWYZg3\nbx7mzZuH4uJiRUDMZjMOHjzosW6NjY2oqanRhIg6OjpgNps1zsRisQCA0wF3ZWVl2L17N2pqavDs\ns8/io48+0pwLAIODgx7rIvnggw/wwQcfeH08wzD+CQuDBx577DHccccdbo9JTU1FYWEhYmJiEBBw\n6pGqw0ktLS2YOXMmHnvsMQDWUNLcuXMRHx+PoKAgZeW5P/3pT7jnnns81quxsRHDw8OaHk3t7e0Q\nQqCnp0fZZrFYEBkZ6VIYNm/ejBdeeAHbt29HUFAQmpublf0HDhzQTBviiaqqKkWIGIY5fWFh8MBN\nN92EGTNmuD3GaDTi2LFjShhJMnPmTGU0dUtLC372s5/h8OHD+M///E989dVXyliKefPmKeGk3bt3\no6WlxWO95NoRcgwFYBUGAJrG2Ww2Y+nSpQ6hpK6uLnR3d+Oee+7Br371K2zevBlz587VXO/FF19E\na2urx7pIqqur0d3d7fXxDMP4JywMk0BqaqpTYUhJSVEa8JaWFqSnp+PPf/4zhoeH8f3vfx/p6ekA\nrOtRf/bZZwDGJgxJSUmorKxUtklhUOcZLBaL07mbysrKkJOTgzVr1uCcc87BXXfdhYyMDEUYurq6\n8M4776C3t9dtPX7605+ipKQEAAsDw5wpsDBMAkajEc3NzUpXVfV2mWNoaWlBYmIiVq9ejV/+8pd4\n/vnnlbDTVVddhbfffhu1tbXo7+9HZ2enx+6ljY2NOPvssx0cQ0BAgEYYzGYzFixYAJPJpAkxSWEg\nInz++edYsmQJMjIyUF1dDQB48803cdFFF6Gnp8ch76Bm586d+Pzzz9HT04Pm5mYWBoY5A2BhmARS\nU1MBwKNjSExMdHr+ihUr0N/fj9/+9rc499xzERMTg7a2NrdlNjY2Yu3atQ6OwWg0OjiG6OhoZGZm\narrPSmFQo3YMr776Km677TYEBga6TUC3tLTg6NGjqKmpQXBwsMscw9NPPz2ubrkMw0w9LAyTQEpK\nCoDxCwMR4eqrr8azzz6Lc845B4mJiUo46frrr3dYKnR4eBjt7e1YtWqVg2OYNWuWg2MwGAzIzc3V\nhJPk7LBqZs+ejaqqKnR2dqKkpAT5+fnQ6XQuw0mDg4Po7OzE0aNHUVVVhXnz5rl0DG+//TbeeOMN\nzbaOjg6MjIw4PZ5hmOmDhWESCA8PR2xsrEMoSQpDf38/+vr6lPWlnXH11VdjaGhIIwzd3d144403\nlB5LkpaWFsTGxiInJ0cZKzE6Ogqz2Yz09HTNW7vslXTWWWfhxRdfRF9fHwD3jmH37t1YtWoVQkJC\noNfrXQpDa2srDAYDCgsLUVlZiby8PJfC0Nraig8//FCz7Z/+6Z9w++23u3wmDMNMDywMk4TRaHTq\nGBoaGhS34G5thmXLluGxxx7DokWLFGGQo5Xtw0qNjY1ISUlBbGwsAKtT6OrqgsFgQExMjFPHcPfd\ndyMlJQWbN29GW1sbysrKkJ2drbnujBkz0NLSgh07dijrYeh0Ok1uQk1zczMyMjIQFxeHf/zjH1iw\nYAF6e3sdZosVQqClpQXV1dWaKTvKy8vx9ttv45lnnsGWLVtw//33u3w+Y+Ho0aOTch2G+bbCwjBJ\nZGdnIy0tTbPNYDAgMDAQFRUVLsNIEiLC/fffj8DAQCQmJqK5uVkRBpPJpDlWCgMRKW/57e3tiI2N\nRWRkpEOOwWAwICgoCK+99hoWLlyI3Nxc6PV6ZTCeJCgoCGlpadi2bZuyQJG7UJIUvMWLF2Pnzp3I\nyspCWFiY4kokUlg2bdqkuIbR0VHU1dXh/fffx1NPPYXe3l4cOHDA7TPyBiEEVq1a5fDMGIbxHhaG\nSeKdd97BhRde6LA9JSUFX3/9tUdhUGPvGFwJAwBkZmaisrIS7e3tiImJgcFgcBCGyMhIANYV7J55\n5hns3LnTYZ4lyezZs9HX14ezzjoLANyGkqQwLFq0CIODg5g9ezYiIiIcEtCtra1ITEzUCENTUxOi\noqKwevVqNDY24sEHH9QMrhsv7e3tGBgYQGdnp8djr7nmGnR1dU24TMn+/ft5skLmjICFYZIIDAx0\nGipKSUnB0aNHfSYMOTk5KCkpceoYRkZG0NfXB71erzl/yZIluOmmm5yWnZGRgbPPPltZ9tSTY0hK\nSsLixYuVcyMiIhzyDFJA1q1bh127dmFkZAS1tbWYNWuWckxycvKkCIPMx3jT4H/88cea0FZfXx+W\nLFmi6ek1Fn70ox/hq6++Gte5DONPsDD4GKPROGbHkJSUpAhDTk6OIgz33nsvnn/+eY0w5OXlobCw\nUCMM8o29u7sbERERbnMb9lxyySUa0XCXY5AN/tKlS5GSkoKoqCiXwpCQkIDY2FjEx8ejqqoKNTU1\nmDlzpnJMfHw8Ojo6JvzGLXuBeSMMPT09mvzNH//4Rxw9ehTvvffeuMq2WCweBwQyzOkAC4OPSUlJ\nQVFR0bgdw7JlyxRhKCwsxOOPP47q6mqNMBw7dsypY5CJ57Hwve99D1u2bFF+2zuG5uZmZSW75uZm\nJCYmYubMmUpXWIPB4CAMMpQEAAsXLkRhYSFqa2s1whAYGIjY2NgxTcHhDG8dw9DQEIaGhpTR4oOD\ng3jyySfxi1/8YtwTAbIwMGcKLAw+JiUlBUNDQ5MiDPX19YiIiMCOHTsUYcjOzkZ9fT1OnjyJ2NhY\nTY5BnV9yaMnBAAAgAElEQVQYL/Y5hpKSEuzduxcdHR2asRkRERHKf+1zDOrjFixY4FQYAKtTmmg4\nyVvHIO9JCsObb76JuXPn4r777sPBgwfHlXvo7u5mYWDOCFgYfIzRaASAMQtDY2MjTCYT8vLyNMIg\n16uWwhAUFIS5c+di9+7diImJmbBjsMfeMcgpMwoLC50O2nMXSgJOOYaamhpNjgGYnDxDU1MTiMhr\nYZChpH379uG73/0udDod1q5di48/HtvS5IODgxgcHHTokcUwpyMsDD5GNuBjEQb59m00GpGUlAST\nyYT+/n5YLBZs3LgR27ZtUybgA6zhpAMHDjiEkibDMdjnGKQwHDt2TEk+29fdVfIZsArD8ePHXToG\n+8F8Y6WxsRGzZs3yKAzynqRjaG5uVu5l06ZNeP/998dUrrxnbxwD91xi/B0WBh8zHmEgIiV2Hx8f\nD5PJhPr6ehiNRgQEBODaa69FYGCgcnxeXh6Gh4cdks+T4RjsQ0lVVVVYvny54hikE5A4EwZ1jiE3\nNxe1tbWorq72SSipqakJubm5Y3YMamFYvXr1mHsXjUUYNmzYgD//+c9jur4zamtrsXPnzglfh2Hs\nYWHwMVIY7BtQT0hhiIuLU4RBTtZnT15eHgD4zDHYh5Iuv/xy7N69G6GhoQgPD9cc7yz5rBaQ4OBg\nZGdnIzw83KFukxFKamxsxJw5c8bsGNSuJjo6eswLDsnjvRGGlpYWPPDAAxOeJ+rDDz/Eiy++OKFr\nMIwzWBh8TFRUFN5//32HBtQTUhhCQ0MRGhqK4uJir4QhPDwcg4ODGBoaUkY9TwRnoaTNmzejuLjY\nqQvylHwGrOEke7cATL1jCAgIcBpKsh897g1jcQw9PT3o6urC22+/ran3WIWivr6epzlnfMKUCAMR\nBRDRESIaXwfx0xgiwqZNm8Z83vz587Fo0SIA1j7+R48edbmSXEJCAtauXQuj0QgigsFggMVigdls\nntReSf39/TCZTFiwYAFmzJjhUhjUjZUQAq2trRrHtGDBApfCMJEcQ19fH3p7ezFr1iyPI597enpg\nNBrR1tamrDkhczuyZ5e7dSjsGYtj6OnpwcMPP4wnnnhC2bZlyxZ8+umnmuP+/ve/u53ag4WB8RVT\n5RjuBFA0RWWdETz11FP43ve+B8AqDF9//bVLxwBA6ZUEnHrjnSzHIBu72tpapKWlITAwEHl5eV4J\nQ1dXF8LDwxEWFqZsu/7663Hfffc5nDtRx9Dc3Izk5GRER0crjmHPnj2oqKhwOLa3txdpaWlob29X\nxmPIgYAhISEIDg722MPIbDYrjftYHcPFF1+sGWHd1tamdLWVPProo9izZ4/L67AwML7C58JARDMA\nbADwB1+XdaYSHx+PY8eOuRUGNTIBPdndVaurqzF79mwA1nCQN8LgLEE9Y8YMrFy50uHcseYYRkZG\nkJ2drdSvqakJycnJiIqKUoThN7/5jdMeRj09PRphsO9d5U04qbi4GC+88AIAqzAQkUdhEEKgp6cH\niYmJ6O/vV3oomc1mhyVd29ra3IbEvBGG/v5+t/udUVRUpKw8yHw7mQrH8AyAfwPgvS9nNMTHx6On\np8dlKMketWOYzO6q1dXVyMjIAADceuutuPPOOx2Ol2EsibpHkifGOi3GyZMnUVFRoTRicqoQtTDU\n19c7DSv19vYiISEBw8PDOHHihEMdvRGGlpYWJUdhsVgQFxfnURgGBwdBRAgNDdUMRuzq6nIQBpPJ\n5LYO9fX1LqcrAazCOWPGjDGPrXj66ac1+Y/poLu7G3fccce01uHbTJAvL05EGwE0CyGOElE+AJeT\n9jz44IPK9/z8fGU9AObUynDeOgbZ4Ex2d1W1Y7AfnCZx5hi8FQb1tBgpKSn48MMP8dZbb+H11193\nenxpaSkAqyBkZWUpjkEdSqqrq3MqDD09PdDr9YiLi0NxcfG4HENrayv6+vrQ19eH7u5uJCYmehQG\nWa66DLmGhloYRkdH0d7e7rIO3d3d6O7udpuwbmtrU0JUUtC9oaOjY8y9siab8vJyvPDCC3j88ceV\n3M+3nYKCAhQUFExJWT4VBgBnA7iMiDYACAdgIKLXhBA32B+oFgZGixQGOYraE5PtGGRjV1VVhWXL\nlrk93ptQkjtkOEnOMbV7926Xx8r5mWRsXjqG8PBwDA8PY2BgAA0NDS4dg16vR2xsLEpKSpCZmanZ\n761jAE41pGMVBulsZEhJLQydnZ3KqnzOqK+vx8yZM1FTU4PR0VEEBDiafxmWa2hoGJMwuBMkZzzy\nyCMAgMsvv1zpIeeJ1tZWjIyMIDk52en+hoYGCCHwzTffOA07fhuxf2F+6KGHfFaWT0NJQoj7hRDp\nQogMANcC+NSZKDDuiY+PR2JiojIVtid8lWOwnyrbGfbCIN/ivUXdM+nkyZOora11WMFOUlpaipCQ\nEI0wJCcng4gQFRWFyspKZV1qe3p6eqDT6RAbG+u06623jgGwNqTeOgYpSMApYZDuRi0M8p5d5Rjq\n6+uRlpaG8PBwl2VKYbBPanuivb3da8cghMAvf/lLNDQ04Morr/S6jP/3//4f/uu//svl/vr6egDW\nEfbM1MPjGE4D4uPjvQ4jAdYwz5NPPomKiopJ6a4q49juBtlJ7HMMTU1NyiA/b0hLS8OJEycAWIWB\niFwu1VlWVoaVK1cqOQb1/EtRUVH45ptvAMBjKKm0tHRcoSTZkMuGdLyhJLPZjNDQUI0wyG6q7hxD\namqq05HmErVjGAtjCSU1NjbCYDDgueeew4kTJzA4OOjVeV1dXW5n0m1oaEBUVBQLwzQxZcIghPhM\nCHHZVJV3JnHWWWfhn//5n70+/v7778fLL7+M22+/3el4gbEgHcPIyAhMJpNDA2rPRB2DXJEOsArD\nypUrXU5PUVpaivz8fOWNWJ0DiYqKQlFREVJTU12GkqRjGBgYGLdjCA0NHZNjcBZK6urqQmZmJlpa\nWpSxE21tbSAir4TBVQJaTig4HsfgbSipqqoKmZmZCAkJQVpaGqqqqrw6z2w2u3SCgFUYLr74YhaG\naYIdw2lAeno6br31Vq+PDwgIwLnnnotHHnlEM35gPMhQRXNzM2JjYxEcHOz2eCkko6OjAE6Fd7zF\nXhguu+wyp8LQ19eH5uZmrFq1Co2NjRgZGcHJkycVIZSOYf78+ejo6HA4X+0YAIzbMWRnZyvCkJSU\nNG7HkJycjODgYOVN3WQyIS0tbcKOIScnx61j2LVrF8rLy5XffX19yoSN3lBZWankL3JycpQOAZ4w\nm81uB+81NDTg0ksvxbFjx8Y00NCX9Pf3K3mtMx0WBsYtQUFBCAoKQk1NjVfJ78DAQE3ce7yOob+/\nH52dnVi/fr1GGDo6OlBTU4Py8nJkZGRgxowZaGxsRF1dHRITExEaGgrglDAsWLDAo2MAHCc5VAvD\n9u3bnTb4ra2tmDNnzrhDSdIxyBHqch0OwOoYMjIyNMJQXl6Om266CQcOHFCEQa/XuxWGJUuWuHUM\nTz31lKZrqhRRbx1DZWWlkrjPzc31WhgsFovHUd2LFi1CWFiYZvnVsdDU1DSpS61+8skn+Jd/+ZdJ\nu54/w8LAeESv16O8vNzrXIF8ixVCjEsYKioqUFdXB6PRiAULFqCmpgZ1dXW45ZZbkJGRgTVr1uCL\nL75Abm4uUlJS0NjYiKqqKiWMBFgb3bKyMsyZMwe9vb0OYyNkAx0bG4vAwEDFOUjUwnDnnXfir3/9\nq2a/EAItLS3Izc11CCW5e8N1FUqKiorSCIPJZMLs2bOV5POuXbuwevVq9PX14c4770RdXZ1XjmHJ\nkiVuHcPx48c1jXl7eztCQkK8dgwylARYHYOzN+qRkRGH5+9NKCk1NVVZunY8/P73v8dTTz01rnOd\nId3dtwEWBsYjOp0OFRUVXneXlQnorq4uhIaGQqfTeV1WTEwMQkJCcOTIEaSlpSE4OBhz585FXl4e\nQkNDUVpaiptuugn/+q//ipycHMTFxaG7uxslJSUOwjA8PIy0tDSnYSHpGOLi4pCQkODQ3VOeI7uR\nbtu2TbPfYrEgJCQEqampSrI2NjYWRIShoSGX9+cqlOTJMRw5cgTXX389tm3bhr6+Phw6dMhrYXDl\nGDo7O1FXV+cgDO5CWPaoQ0muHMOTTz7p0B3dbDajs7PT6bMaGhpCR0cHEhISkJeXh6+//tqrutiz\nZ88etwMAx4rFYpn28R1TBQsD4xGdTofKykqvhUE2VmPNL0gyMzNRUFCAtLQ0AMDPf/5zvPrqq3j+\n+eeRmJiIBx98EIsWLcKSJUsQEBCApKQk7N27V9NXPyoqCoB1+o3o6GiHcJLaMTgbgCcb7ebmZkRH\nR2P37t3KKGfg1PiM2NhYxTFEREQ4TFO+e/duTSMru8nKOkrHIIVB9iSSjkFO5icHCgYEBCjzM6Wk\npHhMPs+dOxc9PT1Op8b45ptvkJGRgbKyMsXldHR0YObMmWPKMXhyDE1NTQ4TBMpnon6mksbGRiQm\nJiIwMBCzZs3CyZMnnZb98ssvK91a7RkZGcHevXtZGMYJCwPjEb1ej4qKijGFksxm85jDSJLMzEzs\n2rVLWaVuy5YtuOyyUx3aQkJCsGfPHlxzzTUArA3k3r17HRwDYB0tHhMT4yAMcjzBmjVr8NJLLznU\nQQpDQ0MDZs+ejXXr1uEvf/mLsl9O9SGFwWKxOBWGa6+9FrfccovS8No7BpljsA8ltbW1ITU1FQEB\nARgYGNCMIF+3bh0KCwsREhLi0jGMjo4qvciSk5OduobCwkLk5+eDiDRjMmbMmIGBgQGPU5NYLBb0\n9PQof2Oj0Yienh6HZ93V1YVDhw5ppuawWCxISEhwGk6SYSRA213anldffdVlDuH48ePo7u6eVGHo\n7u5mYWAYiU6nQ3l5udeOISsrC6WlpWMewyDJzMxESUmJ4hicERgYqMyGmpKSgqqqKgfHoNPpEB0d\n7dIx6HQ6hIaGOh1ZqxYGo9GIa6+9Ftu3b1f2qx2DnLrbXhjq6uowMDCAwsJCZVoP+xyDq1CSyWRC\nXFycIh7qZVSJCPPnzwcAl8nntrY2REVFITg4GEaj0akwHD9+HAsXLtSEgNrb2xEXF+d0XQ17ZF5H\n/h2IyKlrMJvNGBoawoEDBwBY8zNmsxmzZs1ymoCWz1zen6uEfl9fn8uQ1xdffIFly5ZNumOQubMz\nHRYGxiM6nQ6dnZ1eC8OyZctw+PDhCYWSALgVBjVSfOwdQ2pqKojIqTCoRyA7QwqDnGZjzZo1moF2\nasdQX1+PkJAQBAUFQafTKW/GBw4cwKpVq/DKK6/gF7/4BQDXyefIyEgkJSVpHIMUBjmPkjez2UrU\nM8ampKQ4TUAfP34cCxYsQG5urtKYd3R0ICYmRhk939/fj4MHDzp9RuowkiQ3Nxfbt2/HCy+8gIGB\nAQBWx7BkyRJlepP+/n4EBgYiNTVVIwxHjhzBxx9/7CAMrhp3d91q9+zZg0suuWTShWF0dNSrqdVP\nd1gYGI/ImPhYhWG8oaSsrCwAYxOG0NBQjTuZMWMGcnJyAFiX6lSPZRgdHUV/f7/bMR72jiEpKQl9\nfX2a6SukY2hpaVGmHlE7hv3792PFihXIy8tDU1OTMuW2p1CSEGJMwuCs8VMLgzPHIITQCIPaMcTG\nxioTMRYUFOCnP/2p02ek7pEk2bBhA44dO4atW7eiuLgYgFUYNm7ciC+++ALAqSVn5Xrmkr/85S+4\n4YYbUFZWpoSS7FcQVKN2DDt37tRMsbFnzx6sW7dO+VsIIfC73/3O6XW8RYrQtyGcxMLAeESv14OI\nvJ4ldfHixSgqKkJtbe2UOAaj0YhZs2ZpehatWbNGWYfB3jH09fUhPDzc6cRzEnvHQESawXfSMURG\nRiIwMFCZAVQtDAcOHMDKlSuVtbG7urqchpLUyeeWlhaYzWaEhYUhNDRUcRWuJiN05RiamprcOobm\n5mYQEZKSkjQD0zo6OpS1wy0WC5qamlw2hEVFRcjNzdVsu/7667Fz504sXLhQEeOuri5s2LAB+/bt\nw8jIiDKHV1xcnCbH0NLSgp6eHrz00ktjdgyFhYXYu3cvAGtvq/b2dixatEg512Kx4Cc/+Yky8HI8\nyOfMwsAwsDZ2SUlJCArybjJenU6HzMxMfPrpp+PKMaSkpOChhx5SBp95Ys6cOVi+fLnDdhn7thcG\ndc8gV4SFhWFkZEQzsE8tDPINnogQExPjIAwjIyM4fPgwzjrrLADW5VdbW1udOgYpDMnJyairq0N9\nfb0yo25kZCQaGhoQGBjoNPTlTSjJmWOQo8KJyMExxMTEKF2Om5qaXMbxDxw4oNyfPTIpD1iFISMj\nA0lJSSgpKVFyKvaOoaWlBQ888ABGRka8Ega1Y+jq6lJERibdIyIilHElsjEf69oUatgxMIwKnU7n\ndRhJsmzZMphMpnE5BiLCAw88oDTsnli7di3eeOMNl/vthcFTfkHWITIyEqWlpYq42TsG+QYvG1Lg\nlDAUFxcjOTlZETcpDOqy5RKiTU1NiIqKQnx8PNauXYunnnpKGXAXGRmJ8vJyl25NJp9HRkawadMm\npSeRvTDYjx5WJ+uzsrJQU1ODoaEhh1CSK2GwWCyorq7GwoULndZLLQwyVJaeno6Ghga3wrBmzRr8\n9a9/xdlnn63cnzeOobOzUyMM8fHxCAwMRHBwsOa4ieQcLBYLwsLCJl0YysrKvJ5jaqpgYWA8otfr\nxyUMAMYlDJPNeBwDYG2UT548qXEMcv3opqYmpbGOjY11cAwHDhzAihUrlGs5cwyANZzU1tamzIL7\n0EMP4bXXXtM4hoqKCpfCIB1DQ0MDPvzwQ2XeI3UoKT093WEsQG1trTKvVFhYGLKzs/HVV1+5DCXZ\nh2AOHz6MRYsWuZw7KyYmBu3t7cr4ibCwMCQnJyvXk8KgDiXJtbc3bdqkPCNXwiCEcHAMUmSkMKjP\nl8dNJHHc3d0No9HoURi6urrwwQcfeH3dl156Ca+++uq46+ULWBgYj+h0ujGHhJYvX46AgADlf9Dp\nRI5jaGhowO9//3uvHANgbZTVuZWsrCxUVlais7MT1dXVmDt3LgDnwlBUVIQFCxYo10pISFBi6Oqy\npSDI/y5fvhybN29WHENUVJRHYejp6UFNTQ0AKNNHfPPNN0r90tLScPLkSU03S/u1NS666CL84x//\n0ISSpGMA4BCushc+e2JjY9HR0aFM9wFAEQZ1jsHeMdjfp6vuqnJ6b/tQkhBC6eorz+/p6VEa84kI\ng8Vi8UoY9u3b5zDSWwjhdDJHwOp2/C08xcLAeGTDhg3YsmXLmM5ZsmQJ7rvvPgQGBvqoVt4jHcOf\n/vQnPPTQQ2NyDOrcigwl7dq1C2vWrFF6NcnQC3BKGOy7crpzDHq9XvOcnn/+edx1111KHbxxDLW1\ntQCswjAwMICioiIsWrRIOSYsLEzzdq52DABw4YUXYufOnTCbzYiOjtY4BsBxUj1vhKG9vd2lMNiH\nknp7ezE0NOSwfkhYWBgGBwcdljCVTkQdShoeHlZmbbV3DN6Ekvbt2+d2Yj9vhaGtrc1hgaV9+/bh\n0ksvdXp8Z2en383BxMLAeGTZsmU477zzxnROWFgYHn30UR/VaGxIYfjggw9QX1+P6upqrx2DOoSW\nlpaGlpYWfPDBB7j44ouV7c4cg70wJCYmuhQG+8Zw5syZSjI9MjIS7e3tHoWhpqYG2dnZOHbsGAoL\nC5Gdna0Rv/T0dGUBJMBRGM4991zs27cPERERCAwM1CSfk5OTHRqugwcPuhUGGUpyJgzOuqvKXl72\neSUicrpKnUwiqx0DYG2U3QmDK8fQ0dGBTZs24d1333W6X3Y1Tk5OHpcwFBcXu5y+w5ljGB4e9jjy\n3JewMDBnPNHR0Th58iSOHDmC1atXY/fu3V4LgzqEFhQUhLS0NLz11lsaYUhMTERMTAyAU/3u7fv4\nqx2DusGOjIxUGk5XdZBlOEMmn2tqarB582YUFhbi0KFDDr201MIwPDyMpqYmzJgxQ1PO0qVLlWR5\nZGQkWlpa0NfXh/T0dI0wyMbdfgyDGk+hJHnfPT09GBoaUvILru7R/k2/v78fAQEBGscQHR3tIAzy\n7+FJGB5//HGYzWYlYT48PKxxWL29vQgNDUV0dLRHYTCZTA5CWlFRoVmISY0zx/D0009P6sywY4WF\ngTnjiY6ORldXF84991ysXbsWu3fv9jqUZJ90z8zMREREhKY3zl133YV77rkHgLUhqq6uRlhYmKbB\nT0hIQHNzMwYGBhAeHq5sd+YY1MhreBNKuvDCC9Hc3Ixdu3Y5CIN6ydT6+nokJiY6JI4vuugiRRgM\nBgPKy8uRnJysjLeQfP3111iyZInbXmMylCR7JAGOOYaAgADlOFcD+ADnwtDX14e4uDiNY8jMzITJ\nZBpzKKm2thZ//OMfceuttyrC8NFHH2lCPxaLBQaDwWHpWme0tbVhYGBAGfkNWNfSGBwcdHquM2Go\nra3VLPU61bAwMGc8Moa/ceNGLFu2DBUVFV45hpiYGIc1rrOysnDRRRdpBsfp9XpNjuH48eMOb9MJ\nCQmora11GFjnSRg8OYawsDAMDQ2hoqICmZmZmDt3Lt59912njkH2TLIPI0muuOIKJWRoMBhQVlaG\n5ORkh2nLS0pKlMS2K9Q5BnkP9o4BAOLi4tDa2qqZC8oe2bh3dnYq4av+/n4kJibCYrFACKGMlXAV\nSnLXK+mzzz7D+vXrMX/+fEUYmpqacPDgQaUb6ViFAYAmnCR7szlr7J0JgxzoOF14N2KJYU5j5CCu\nTZs2KW9x3jiGe++912F09G233eb2nPDwcJSWluLqq6/WbJfCYB82mmgoiYgQERGB2tpapKenIy8v\nD8eOHUNeXp7muPT0dBw5cgSAa2FYvHgxFi9erJTb2tqK1atXOxUGOYmfK5zlGGJjY5WV2+R95eTk\noLi42KNj6O3tRVNTE44fPw7A6hgMBgOCg4NhMpkQGhqK5ORkp46ht7fXrWNoaWlBSkqKZuyFyWQC\nEeGdd97BvffeO25hSExMhBACFRUVmDt3LlpaWpQpXwAoomb/77GlpcXtyHxfw8LAfCs4fvw4iAhC\nCKUnkCdk3kCNpwZRp9NhZGTEqWMYHBx0KHeijgGwhpPCw8MRHh6OhQsXYuHChcoSpxJ1jsGVMKiR\nDigpKQk6nc5BGK644gq350dGRqKvrw8mk0kRhoCAACQmJqKiokLTPffQoUMYHBzU5DzUyLd+OXZh\nZGREmetKjjWJiopSpthwFUqKjo526hhkfiMuLk4jDBs2bMDbb7+Ne++9V1lvwxthMJlMCAwMVBxD\nU1MTdDodsrKylOnNJXJ1QftrtrS0jGmBq8mGQ0nMtwL11NBLlizx2f908rr2whAeHg69Xu8gDJs3\nb8bNN9/s8nqyAXU3HiQiIkIZk/Dd734XW7dudThGnWOoqanxKAzq8I8zxzBnzhy358upQmpqajSO\nKDk5GZWVlYrwLF++HIcPH3YbSpIJZDkOoLe3V5nvymAw4MSJE4iOjkZ8fDxaWlrQ2dmpiLpaGJKS\nkpwKg3Qrcgp1wNq4X3HFFairq0NFRcWYHYM6YV9RUYHs7GzNtOqSzs5OTUhMXafpHNvAwsB867j2\n2muVPv6TjRQG9doQkoSEBAdhyMnJUaZ/cEZcXByee+45t/NU6fV6paGfNWuWZlEjidFoRGtrK4aG\nhsbkGOyFoaurCxaLxSH34gxXwjA8PKwIj5yJ15teSVIY5Ip0zhxDRUUFoqKilOelFobk5GS3M9Gq\nQ0mtra1ITk7Ghg0b8Mknn4xZGDIyMhTHUF5ejqysLGWQo5rOzk7ExcUhNDRUEa2BgQFl1t3pwqfC\nQEShRLSfiL4iokIicnyVYZgp5pZbbsH69et9cm1XjgFwLgyeCAgIwB133OH2GLVjcEVQUBCSk5NR\nX1/vlTC4cgylpaXIzc31Kv4dGxvrIAzSFcjrJyQkIDIyEgcPHvRaGLq7uzWO4eTJk4iOjkZcXBxK\nS0s17kqdfPbGMahDSfHx8cjJyUFlZSW6u7thMBg8LmDU19eH4eFhGI1GRRgqKiqQlZWljGVRI7vZ\nqp9xa2srAgICzlxhEEIMADhfCLEEwGIAlxKR61ExDHOao9PpEBYW5nQKkfEIgzdERER4bOgBazjp\ngQceQEdHh0chkVOt2wuDN2EkSWxsLBoaGhwcAwBNXmX58uUwm81jEgZnjiE+Pl4zM636XOkYXOUY\n5GysspupFIaMjAxUVVUpS7d6cgxyHQ05XTrgOZRkLwwtLS1IT093KOfVV1/FCy+84LLsycTnoSQh\nhPxLhMKa7D7z18VjvrWkp6c77c0EWBPIvhCGq666Cueff77H43JyclBeXo79+/drxlI4g4hgMBgm\nJAwxMTEQQmhEQAqDHCkOQOla62y9CeBUzyL7UFJ4eLhDKAmAW2GwDyUJIZSZcolIGZjnTBi8CSW1\ntbUhPj5eM/bDUyhJPQUJYBWGzMxMh7xDSUkJSkpKXJY9mfhcGIgogIi+AtAEYKcQwvk6gQxzBhAR\nEeEwgZrEV47hBz/4gcdxBQDw4osvYs+ePU7zH854/fXXMWvWLAdhsF+cxxVysJy9Y4iIiNAI5/Ll\nyxEbG+typlbZuMswjwwlhYWFwWAwoK6uTkk+A1AEQn2uq+RzZ2ensva3rHNDQwMGBgZgMBiU+bGk\nMOj1eqVnlJqRkRGMjo4qE/ipHUNNTQ1mz57tNpQkJy0ErKGklJQUhIWFaYRMrt0xFfi8u6oQYhTA\nEiKKBPA3IponhCiyP079P1N+fj7y8/N9XTWGmVKMRqNDgzKV2Hdh9YRMYk8klAQ4CoN999zVq1fj\nvvvuc3kdvV6PtrY2dHR0IDAwUOMYAgICUF9fj6ioKCWM58oxOBMG9boVgFVUysvLER8fr/SsIiKc\nOHECs2fPRkBAgDINibwvIQQ2bNiAyy67DAkJCYowlJWVobe3F/39/YiNjcXAwIDXoSS5OqDZbFbc\nVRV7NhYAABChSURBVFlZGerq6ly+eEwmUzaOQQhhJqICAOsBuBUGhjkTueWWW6Z1YrTxIhuo4eFh\nVFVVITs726vznAlDTk4OLrjgAs1xERERuPvuu11eR91d1Wg0Ko5Br9cjJCQEw8PDShlxcXEaYdDp\ndGhra0NgYCBiYmIcQkn2A+tiY2NRVlamXIOIkJGRga+//hqbNm0CACWcJMvcvn07CgoKEBkZiQsu\nuEAJJXV1daG+vh5GoxFEpEwaODo6qjimzs5OJQHvTBjUYauwsDAYjUalrXzooYc8/QnGja97JcUT\nUZTteziAiwBMTZCMYfwM2YvmdEM2WtXV1TAajR7zE5KYmBgEBgZqxowkJSXh9ddfH1P56uRzWlqa\nQ/IZsM6HBVjzC/aOoampCQaDQbMet8TeMdgLA2DtelxWVqb87dR5BovFgrvvvhsvvfQS9u/fr4SS\n5LKt9fX1StfekJAQREREaBaN8sYxSMxm85T1VPJ1jiEFwC4iOgpgP4AdQoiPfFwmwzCTiIx/FxcX\nex1GAqyNbFRUlNdLtLrCXhh6eno03VWBU64kKSlJ09Dr9Xo0Nze7FAZnjsG+y2tGRgaEEE6F4dNP\nP8XcuXNxww03oKenB8ePH1dCSWazWSMMABx6JnV1dSk5BnXyOSEhQZN3kMdOlTD4NJQkhCgEsNSX\nZTAM41uCg4MREhKCr776yuvEM3BKGCaKuleSJ8fw+uuvKyEsee7Q0JCSOLYPJblyDOpJCGWyXsb6\n1Y14TU0NcnNzQURYsWIFPv74Y2zcuFETSlILg+yZJAV2LI6hq6tLWaDI1/DIZ4ZhPBIZGYkDBw6M\nyTHk5ubihhtumHDZer0eHR0dGBoaQmJiolvHkJiYqBklLnuBRUZGahyD2WzG6OioU8fQ1dWlcQxy\nsKIzx6BeInXlypXKSGZXwmDfM8mTMKhzDFPpGFgYGIbxyHiEIS4ublI6lej1eqVLqsFgcOoYXDkT\nKQwylCQdww9/+EM899xzToUBgEMoSV5D/lftGKQwrFq1SrlvKQx1dXUOwnDkyBGlE4K9MAghlFCS\nWiyEELBYLBgaGlLWu/YlLAwMw3gkMjISJpNpTMIwWeh0OjQ3NyMmJkbpKmrvGGQoyZ7Q0FAEBATA\nYDAgJCQEo6OjynxRzz33HBoaGhxCSYBWGNLT0xEUFKSZJkQu06kWBrlWRHx8vDLe4cSJExph2LJl\nC3bu3IkZM2bgxIkTDuMYuru7ERgYqKzxIYWhu7sb4eHhDosm+QoWBoZhPBIZGYno6Gi303/7Cr1e\nDyEEYmJiEBER4TCJnlyTwhlEpDSyRASdToe+vj40NjYiICAA+/fv19yTs9HTwcHBOHLkiOJKFi5c\niMLCQgBaYYiNjcUjjzyCtLQ0RYxKSko0wnDOOefgwIEDuOqqq/Dyyy+js7NTmXrdYrGgublZGQFu\nP3mhPI6FgWEYvyAyMhJz5syZcA+j8SDDQWrHIAe4xcXFYe3atW4n9dPr9crbvk6ng8ViQUtLCx55\n5BEA8OgYAGiWcpXC0NXVhcHBQc1I61/84hfKQMKoqCh0d3c7LA8LADfffDNeeuklBAYGIjQ0VGnw\njx07hnnz5gGAJscw1cLAC/UwDOORqKgol+EaX6MWBrnGtZwSQ6/X4/PPP/d4vgw56fV6ZSW9a665\nBiUlJZqR2K6EQc28efNQXl6OiooKzJo1y6VYRkVFISEhASEhIQ77Fi9erBEM2eB/8cUXyjTs6lCS\nXA41ODiYHQPDMP5BfHw8FixYMC1lBwcHIzg4GLGxsZpQkrcD7ezX5K6srERKSgqCgoLwyCOPaBp2\nudSqO2EIDw/HzJkzsWPHDrez2kZFRbldt+Lmm29WxFaKwJ49e7B27VqlLtMVSmLHwDCMRx5++GG3\niwX5Gr1erwklDQ0NISwszOtz1Y6hoqLCaXgHgDIvkqcV/vLy8vD+++9j6VLXw7Q8red94403KvmJ\nyMhIdHR0wGKx4KyzzlK22QtDUFAQOwaGYfwDvV4/5kn4JhOdTueQfPbWMeh0Oo1jqKiocLpehsTd\nGtyShQsXYv/+/W7XtfDkGPR6PTZs2KDUa3h4GHl5ecp9TWeOgYWBYRi/x94xyByDN6xdu1aZllwd\nSpoICxcuhBBiQsKghogQGRmpWebVWY5BzsHkaziUxDCM3yOFQXY3JSKvhUE9yE6v1+PLL7/Eli1b\nJlSfvLw8AHArDLfffvuY1t8wGAwaYXAWSpqqJT9ZGBiG8XuSkpIwY8YMBAQEIDw8HAMDAy4X9nGH\nnIbbVY7BW2bNmgWj0eh20SPZ7dRb7r33Xlx44YXKb3thyMzMBBGhubl5fJUeAywMDMP4PR999JEy\nViEiImLc4ylkUnmioaSAgADU1tZOakL+Jz/5iea3TqfD4OAghoeHFcdAROwYGIZhAGgGsEVERGB0\ndHRc15GhnYkKAwCf99KSI7otFgvMZrPSw4mFgWEYxg69Xj/uieQmyzFMFTKc1NXVpfSWYmFgGIax\nIyIiYtzrEshur94mrqebmJgYtLa2KqEkIQQLA8MwjD2yy+p4zz1d3AIAnH322SgoKFCEYXR0lMcx\nMAzD2BMREeH14DZ7dDrdaSUM69atw44dO5QcA49jYBiGcUJERMS4Q0GLFi1CX1/fJNfId1xwwQW4\n7rrr0N/fj8jIyClzDCwMDMOcVuj1+nE7hqVLl7qd38jfMBgMWL58Ofbu3YuwsDAIITA4OIihoSGf\nlsuhJIZhTism4hhOR9atW6d0VZVTZ6jXgvYFLAwMw5xWTMQxnI5s2rQJaWlpyu+pmEjPp8JARDOI\n6FMiKiKiQiK6w5flMQxz5vNtcwwLFizAwYMHld+nvTAAGAbwcyHEPACrAfyEiKZ+NfHTiIKCgumu\ngl/Az+EU/CxOUVBQgMzMTGRnZ093VaYU9cjvZ599VuMgfFKeLy8uhGgSQhy1fe8GUAzAu3lov6Vw\nI2CFn8Mp+FmcoqCgAFdccQXuv//+6a7KtHHBBRcgJibGp2VMWY6BiGYBWAxg/1SVyTAMw4ydKREG\nIooA8L8A7rQ5B4ZhGMZPISGEbwsgCgLwAYD/E0I85+IY31aCYRjmDEQIMb75xz0wFcLwGgCTEOLn\nPi2IYRiGmRR8KgxEdDaAzwEUAhC2z/1CiL/7rFCGYRhmQvjcMTAMwzCnF9M68pmI1hNRCRGVEdG9\n01kXX+FqkB8RxRDRx0RUSkQ7iChKdc7zRFROREeJaLFq+422Z1VKRDdMx/1MFCIKIKIjRPSe7fcs\nIvrSdk/bbDkpEFEIEb1lew77iChddY37bNuLieiS6bqXiUJEUUT0ju0+viGild/ifxd3EdFxIjpG\nRG/a/v7fin8bRPQyETUT0THVtkn7d0BES23PtYyInvWqUkKIafnAKkoVAGYCCAZwFMCc6aqPD+8z\nGcBi2/cIAKUA5gB48v+3d66xVlRXHP/94QpqLaBGsPgAqfrBRzTXCFSa4KNWjQk2sUQTFS6aSDRq\n4gdRmxr1i/FDKxhssTUpJW3qI/jC0JZgMKY14vsRFRXFKKi5iChaPlglyw9rHdlzvAfb67kMh7N+\nycnds2bPzN5rVu6aWbP32sC8kF8L3Brls4DlUZ4CrI7yvsDbwGhgTKNcd/8GoY+rgb8Cy2L7XmBm\nlBcBc6N8GfD7KJ8H3BPlo4AX8ASQE8OGVHe/BqmLPwNzotwT97br7AIYD6wDRhQ2MbtbbAP4KT6U\n/+VC1jY7wKcITI7y34EzvrNNNSpjKj5SqbF9HXBt3TdpJ/T7IeBnwOvAuJAdCKyJ8p3AeUX9NcA4\n4HxgUSFfVNbrhB9wMLASOJntjuEjYFizTQD/BKZEeTiwcSA7Af7RqNdJP+CHwNsDyLvRLsYD78Y/\ntx5gGXA6sLFbbAN/QC4dQ1vsII59rZBX6rX61RlKOghYX2xvYDefFV1M8luN3/R+8BniwNio1kov\nzfL36Tx9zQeuwQchIGl/4BMza6zsXtrAN/01s23AFkn7sXvoAWASsEnS4git/VHS3nShXZjZB8Bv\ngffw9m8Bngc+7VLbABjbJjs4KOo0198hdTqGgcbf7rZfwgeY5Neqr816UdTtaH1JOhvoN0+R0uiL\n+Ha/rNjXTMfroaAH6AV+Z2a9wFb8iber7AJA0hjgHPypeTzwAzxk0ky32MaO+H/tYFA6qdMxbAAO\nLbYPBj6oqS1DSnw0Wwr8xcweDnG/pHGx/0D8tRlcL2WGrIZeOl1f04AZktYBdwOnAguA0ZIadlj2\n6Rs9SBqOx0s/obV+Oo0NwHozeza278cdRbfZBXhodZ2ZbY43gAeBk4AxXWob0D47GJRO6nQMzwCH\nS5ogaQQe+1pWY3uGkj/hcb5y5vcyoC/KfcDDhXwWgKSp+Ot0P7ACOD1GsuyLx2BXDH3T24OZ/crM\nDjWzSfi9XmVmFwKPATOj2myqepgd5ZnAqkJ+foxMOQw4HHh6Z/ShncQ9XS/pyBCdBrxKl9lF8B4w\nVdKeksR2XXSTbTS/PbfFDiIM9ZmkyaHbWcW5WlPzB5cz8VE6a4Hr6v4ANER9nAZsw0ddvYDHTs8E\n9gMejf6vBMYUx9yBj6h4Cegt5H2hqzeBWXX37XvoZDrbPz4fho+aeBMfhbJHyEcC90V/VwMTi+Ov\nD/2sAX5ed3++hx6Owx+QXgQewEeUdKVdADfG/XwZWIKPVOwK2wD+hj/Ff4E7yTn4h/i22AFwAj7J\neC1w+//SppzgliRJklTIpT2TJEmSCukYkiRJkgrpGJIkSZIK6RiSJEmSCukYkiRJkgrpGJIkSZIK\n6RiSjkTS2EjP/JakZyQ9IemcmtoyXdJPiu25ki6soy1J0g566m5AkgySh4DFZnYBgKRDgBlDdTFJ\nw83TNQzEycB/gCcBzOwPQ9WOJNkZ5AS3pOOQdCpwg5mdMsC+YcCt+OzqkXiSurskTQduAjYBxwDP\nmtlFcUwvcBuevG0T0Gdm/ZIew2clT8PzO60Ffo3Pyv0YuADYG599+xWeQvxKPPfP52Z2WyyksgjY\nC8+Rf7GZbYlzPwWcgs94vsTMnmiropJkkGQoKelEjsZTiwzEJXj+mCnAZOBSSRNi3/HAVfiCLj+W\ndFIkOFwInGtmJwKLgVuK8+1hZpPNbD7wLzObamYn4Cka5pnZu3iO/Plm1jvAP/clwDVmdjzwCp76\nocHwaOfVuNNKkl2CDCUlHY+kO/BVsP6LL/hyrKRG8rVRwBHAl8DTZvZhHPMivsrXFvwNYmUkGRtG\nNfvkvUX5EEn3AT/C3xre+Y52jcIzf/47REvwHD8NHoi/z+Epp5NklyAdQ9KJvAqc29gwsytioZbn\ncMdwpZmtLA+IUNIXhWgbbv8CXjGzaS2utbUoLwR+Y2bL43w3tjimcukd7Gu0p9GWJNklyFBS0nGY\n2SpgpKS5hXgffAGSFcDlxcLxR8TKaK14AzggUhgjqUfSUS3qjmL728TsQv557Gtu52fAZkkNp3MR\n8HiLc+/IgSTJTiWfUpJO5RfAAknz8I++W/GY/9LIxf98hIY2Rt1mDMDMvpT0S2ChpNH4GsILgNf4\n9kpXNwNLJW3G1wCYGPJHQj4D//hcHtcH3ClpL3zB+znl9ZvbkyS7AjkqKUmSJKmQoaQkSZKkQjqG\nJEmSpEI6hiRJkqRCOoYkSZKkQjqGJEmSpEI6hiRJkqRCOoYkSZKkQjqGJEmSpMLXzUhMAHm1pIYA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f6af31075f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train loss\n",
    "loss_generations = [i for i in range(generations) if i%output_every==0]\n",
    "plt.plot(loss_generations, train_loss, 'k-')\n",
    "plt.title('Sequence to Sequence Loss')\n",
    "plt.xlabel('Generation')\n",
    "plt.ylabel('Loss')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
